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Abstract. The aim of the present article is to point out the problematic issue of administrating 
PISA science literacy exam to the ultra-orthodox schools in Israel. It has been assumed that 
some texts included in the test may offend the feelings of the ultra-orthodox population or may 
contradict Orthodox upbringing and therefore constitute a cultural bias. 
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PISA Assessment 

The Program for International Student Assessment (PISA) is an internationally standardized 
assessment developed by the OECD in order to examine skills that students have acquired as they near 
the end of mandatory schooling. 

PISA is a paper-and-pencil exam that is designed to assess 15-year-olds' capabilities in reading, 
mathematics, and science literacy. Each student takes a two-hour assessment. Assessment items 
include a combination of multiple-choice and open-ended questions that require students to come up 
with their own response. PISA scores are reported on a scale with a mean score of 500 and a standard 
deviation of 100. Tests are typically administered to between 4,500 and 10,000 students in each 
country. 

The PISA assessment was implemented in 43 countries in the first assessment in 2000, in 41 
countries in the second assessment in 2003 and at least 58 countries in the third assessment during 
2006. Since the year 2000, over 70 countries and economies have participated in PISA. Compatibly, 
PISA 2000 focused on reading literacy and PISA 2003 focused on mathematics literacy. However, 
PISA 2006 focuses on science literacy. 

PISA assesses scientific literacy in three dimensions: scientific concepts, scientific processes & 
scientific situations (http://www.pisa.gc.ca/science_e.shtm). 

PISA was introduced in Israel in 2002, two years after it was administered elsewhere, with somewhat 
disastrous results: Israel was ranked 33rd among 41 participating countries: In reading comprehension, 
Israel ranked 30th. The category was led by Finland, Canada and New Zealand. In mathematics, Israel 
ranked 31st, with Hong Kong, Japan and South Korea taking top marks. In the sciences, Israel ranked 
even lower, in 33rd place. The top marks went to South Korea, Japan and Hong- Kong. 

Only students from Latin American countries, Bulgaria and Albania received consistently lower marks 
than Israeli students in the international survey (Literacy skills for the world of tomorrow, Unesco 
Unit of Statistics, 2003). 
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Picture 1: Average achievements in Scientific Literacy, PISA, 2002 


The aim of this research has been to point out the problematic issue of administrating PISA science 
literacy survey to the ultra-orthodox schools in Israel: It is assumed that some texts may offend the 
feelings of the ultra-orthodox population or may contradict Orthodox upbringing and therefore 
constitute, to some extent, a cultural bias. 

Cultural bias 

Cultural bias is the phenomenon of unraveling and judging event by standards, norms, and 
conventions inherent to one's own culture. When people of a culture make assumptions based upon 
their own upbringing, then they can mistake these assumptions for laws of logic or nature. 

Historically, the research on cultural bias in teaching and learning can be traced back to the research of 
Lev Vygotsky (1978) and Alexander Luria (1976). Both psychologists approach fused "cultural", 
"historical", and "instrumental" psychology and is most commonly referred to presently as cultural- 
historical psychology. It emphasized both the mediatory role of culture, particularly language, in the 
development of higher mental functions and the cultural bias inherent in evaluators' critique of young 
children's problem-solving strategies. 

Cultural bias in teaching is evidenced within various academic curricula, especially in humanities and 
social sciences. Much of the learning chapters throughout those subjects are prepared by members of 
the majority as well as used to reinforce the hegemonic status of this group (Baker, 2005; Loewen, 
2007). For example, the recent complain addressed to the Israeli education system is that the 
Holocaust is a vastly Ashkenazi symbol. Nevertheless, the Holocaust, as it is memorialized and 
studied, under-references the Jews in Northern Africa. Their history is indeed largely ignored, with 
maps, memorials, presentations and other teaching materials and listings of victims largely limited to 
Europe. Lists detailing Jewish losses typically aim to present a comprehensive picture of the losses of 
the Jews of Europe as a direct result of Nazi persecution while Non-Europeans Jews are absent from 
most. 

Some researchers (Watkins-Lewis, & Kizzie, 2006; Tyler, Boykin, & Walton, 2006) claim that for 
many ethnic minority pupils, school-studies are actually conflicting to the cultural-laden learning 
experiences outside the classroom. 

Consequentially, it is particularly difficult to develop an objective test let alone a test that measures 
innate intelligence without introducing cultural bias. Many college students have a middle-class 
background and may have difficulty appreciating the biases that are part of standardized intelligence 
tests, since their own background support the line of common tests. In order to prove this point tests 
have been developed for Blacks and for the Chicano culture and the Redden-Simons Rap Test (1986). 
The Redden-Simons "Rap" Test is a 50-item, multiple-choice test of vocabulary items typical of 
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’’street language” in 1986, Iowa. On the short version of the Redden-Simons ”Rap” test (12-items), 
’’street” individuals averaged eight correct items, and college students averaged only two correct items. 
Using ’’street” norms, middle-class background students failed the test. 

Methodology 

The methodology we applied in our study is content analysis; a series of procedures is used in 
analyzing the text aimed at arriving at significant diagnoses and generalizations from within the text 
(Weber, 1985). Compared to other tools of measurement, this method has 3 major advantages: it is 
not invasive; contrary to other techniques such as interviews, responding to questionnaires, and 
projection tests, it is free of errors in data analysis stemming from the respondents’ awareness of the 
examiner’s presence and expectations. 

Content analysis is able of dealing with puzzling data; while in techniques such as interviews and 
questionnaires the data is obtained in a structured manner so that every category being investigated is 
known beforehand, in content analysis the analyst may not be able to predict all the categories before 
conducting a preliminary check of the text. 

Context analysis is context sensitive; the interpretation of the data in the process of content analysis is 
supposed to follow the processes occurring in reality, including political processes; it is a technique 
that is able to deal with a large amount of data such as data culled from textbooks (Krippendorff, 
2004). 

The analysis of texts can be carried out by using three primary methods: a) an analysis of qualitative 
content, b) an analysis of quantitative content, and c) a combination of both (Gull, 1981). Qualitative 
content analysis is based on understanding the content and the commentary that goes with it. This 
method gives a description of and comments about the principles and the values characterizing the 
description of events in the text. Depending largely on the analyst’s intuition, the method, and the 
resulting evaluation cannot be expected to be objective. In contrast, quantitative content analysis is an 
objective and systematic method describing quantitatively clear messages. A distinction can be made 
between space analysis and frequency analysis; when doing space analysis, the analyst can point to the 
amount of space given to a certain event by counting the number of pages or lines in the text; the 
quantities are reported either in absolute numbers or relatively to other subjects. In frequency 
analysis, the analyst reports the number of times that a concept, a subject or idea are mentioned in a 
certain part of the story. 

The third method is a blend of qualitative and quantitative analyses; in other words, it combines the 
systematic drawing of valid conclusions derived from a text, and is based not only on the 
understanding and interpretation of the researcher (naturalistic generalization), but also on numbering 
outstanding and recurring components in the text itself. In order to classify the utterances and the 
ideas in texts into unequivocal and independent categories, the method requires a detailed system of 
criteria. To balance the disadvantages of qualitative analysis, many passages are quoted from the text 
being evaluated; the purpose of the quotations is to enable the researcher to reach a high level of 
precision in reporting the content (Weber, 1990). 

The extent to which the qualitative and quantitative methods are blended depends on the views and 
inclinations of the researcher; in the present study we will use a synthesis of both with emphasis on 
qualitative analysis. 

The corpus: The sample has comprised texts (67) taken from the PISA science literacy preparation in 
Israel (1st & 2nd files, Published by the Academic Secretary, Ministry of Education, an experimental 
issue, Jerusalem, 2006). 

Results 

It has been found that 9 texts (13.4%) have included problematic issues that may constitute a problem 
to ultra-orthodox pupils. 


Volume 5 Number 3, 2012 




46 


Sara Zamir, Helena Sabo 



Figure 2 : The total sample texts and the controversial texts of PISA research corpus 


The controversial issues found in the research: 

Issues that contradict specific Jewish commandment (ltext: "*About stars and people”). Interpreting 
mystical stars phenomena contradicts the prohibition of star worship, which is regarded in Jewish 
scripts as heathenism and paganism. 

Issues that contradict biblical versions of the creation of the world (2 texts: ’’Alfred Wegner” & ” 
Geological Linguistics”). 

Issues that may seem as intervention in the Devine order (5 texts: ’’Genetically modified plants”,” 
Vitro fertilization”, ’’Choosing the sex of the baby”, ’’Cloned cells” & ” Perpetum Mobile”). 

Issues that contradict the Jewish calendar (ltext: ”*Powers and Cranes”).The text includes dates 
according to the Christian counting of years: Anno Domini. 

Discussion & Conclusions: 

The presentation points out the problematic issue of administrating PISA science literacy survey to the 
ultra-orthodox schools in Israel. No doubt that the texts mentioned above may contradict biblical 
traditions or even constitute a ’’deaf point” in understanding due to the ultra-orthodox socialization. 
Bias is presented when a test score has meanings or implications for a relevant, definable subgroup of 
examinees that are different from the meanings or implications for different examinees. 

The research has shown that PISA test may be at risk to be biased from social values, religious beliefs 
and diverse standpoints. Hence, the concern relates to denial of prospects to do well on the test. Bias in 
item selection is presented when the items and tasks selected are based on the learning experiences and 
language of the dominant group, hence, the secular one. 

One can argue that Social bias indeed influences validity; when a test is shown to measure different 
hypothetical constructs or grasps for one group than another, it certainly impairs the validity of a test. 
According to PISA site (2012) the administrators do follow the potentials of cultural bias of 
immigrants. It has been stated that governments and schools can help students from immigrant 
backgrounds to overcome some of the disadvantages associated with that background. Often, students 
with migrant background are socio-economically disadvantaged. On average across OECD countries, 
the parents of these students are less educated and work in lower-status occupations than their native 
peers. In addition, these students tend to have access to fewer educational and material resources at 
home than their native peers. 

On average across OECD countries, the gap is reduced from 43 to 27 score points when comparing 
students of similar socio-economics status, regardless of whether they are from immigrant 
backgrounds or are native to the country in which they were tested. But the fact that a performance 
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gap equivalent to well over half a school year persists, even after accounting for socio-economic 
status, implies that other factors also have an impact on student performance. 

Another significant concept relevant to the discussion of culture bias is fairness. It can be argued that 
while a test might not be biased technically, it can still bring about unfairness (Cole & Zieky, 2001). 
Test fairness is fundamentally about the social consequences of test results. Test fairness is the extent 
to which the social consequences of test usage are considered fair or unfair to relevant subgroups; test 
fairness is especially important to consider when used for selection, placement and even rating 
conclusions. If a test negatively affects a certain group then it has a incongruent impact. Bearing in 
mind that most male students of that sector do not continue studying secular subject-matters by the age 
of 12, administrating Pisa test in this sector may constitute a cultural bias as well as distort the overall 
achievement results of Israel. Moreover, Less than half of schools for ultra religious Jewish children 
teach English, literature, Hebrew language or Hebrew grammar. The survey, carried out by the Central 
Bureau of Statistics for the Ministry of Education, found that only 38.5 percent of ultra-Orthodox high 
schools and post-secondary institutions teach English. Only 40% teach Literature, Hebrew Language 
and Hebrew grammar (these four subjects together form the core curriculum). Mathematics is only 
taught at 41.3% of high schools and post-secondary institutions, while only 30% of ultra-Orthodox 
schools teach Civics and Geography. Since the paragraphs of PISA, assessment is primarily based 
upon two main dimensions, namely reading comprehension and worldwide education, one can clearly 
learn about the disadvantage of the ultra - orthodox pupils. 

Nowadays, a growing number of educators call for a broader range of assessments, including 
alternative evaluation that can assess higher order skills. It seems as if the current state of standardized 
testing, nationally and internationally, still fails to assess learning readiness or capacity for academic 
inquests. Alternative assessments which take into account performance-based tasks can provide a 
more revealing picture of a students' true competence. Although alternative assessments present 
reliability challenges, an evaluation of a widened sampling of student work can be used to supplement 
traditional standardized test even like PISA. 
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